Skip to main content

The Dataset Collection

The Dataset Collection consists of large data archives from both sites and individuals.
SHOW DETAILS
up-solid down-solid
eye
Title
Date Archived
Creator
The Dataset Collection
data
eye 18,268
favorite 4
comment 3
(Here is the original Reddit comment announcing this collection of data and what the processes were.) This is an archive of Reddit comments from October of 2007 until May of 2015 (complete month). This reflects 14 months of work and a lot of API calls. This dataset includes nearly every publicly available Reddit comment. Approximately 350,000 comments out of ~1.65 billion were unavailable due to Reddit API issues. Q: How are the files structured? Each file is compressed with bzip2 compression....
favoritefavoritefavoritefavoritefavorite ( 3 reviews )
The Dataset Collection
by Gwern Branwen
data
eye 14,311
favorite 3
comment 0
Dark Net Markets (DNM) are online markets typically hosted as Tor hidden services whose users transact in Bitcoin or other cryptocoins, usually for drugs or other illegal/regulated goods; the most famous DNM was Silk Road 1, which pioneered the business model. From 2013-2015, I scraped/mirrored on a weekly or daily basis all existing English-language DNMs as part of my research into their usage, lifetimes/characteristics, & legal riskiness; in addition, I made or obtained copies of as many...
Topics: Tor, Bitcoin, drugs, Silk Road, Evolution, Agora, black-markets, dark net markets
Dumps of DISCOGS.ORG Metadata (2008-Present)
collection
47
ITEMS
10,432
VIEWS
by DISCOGS.ORG
collection
eye 10,432
This is an unofficial mirror of the DISCOGS.ORG data collection, which is located at http://www.discogs.com/data/. Discogs, short for discographies, is a website and database of information about audio recordings, including commercial releases, promotional releases, and bootleg or off-label releases. The Discogs servers, currently hosted under the domain name discogs.com, are owned by Zink Media, Inc., and are located in Portland, Oregon, USA. Discogs is one of the largest online databases of...
MusicBrainz Data Dumps
collection
225
ITEMS
9,908
VIEWS
collection
eye 9,908
The MusicBrainz Database is built on the PostgreSQL relational database engine and contains all of MusicBrainz' music metadata. This data includes information about artists, release groups, releases, recordings, works, and labels, as well as the many relationships between them. The database also contains a full history of all the changes that the MusicBrainz community has made to the data. Core data Artists Name, sort name, IPI, aliases, type, begin and end dates, disambiguation comment, MBID...
The Dataset Collection
by Internet Archive
data
eye 7,304
favorite 3
comment 1
Culled from various sources, this collection includes over one million JPG, PNG and GIF album covers. The resolution ranges from "thumbnail" through to very large sizes. Filenames are variant in usefulness, although a good number indicate at least the name of the original album. This dataset is for experimentation and image processing research only. At 148gb, the collection is large but not unmanageable (there is a torrent available) and allows a developer or artist to work with the...
favoritefavoritefavoritefavoritefavorite ( 1 reviews )
Topics: dataset, big data, album covers, covers, cover art, cover photos
Internet Census 2012
collection
15
ITEMS
3,715
VIEWS
by Anonymous
collection
eye 3,715
Abstract While playing around with the Nmap Scripting Engine (NSE) we discovered an amazing number of open embedded devices on the Internet. Many of them are based on Linux and allow login to standard BusyBox with empty or default credentials. We used these devices to build a distributed port scanner to scan all IPv4 addresses. These scans include service probes for the most common ports, ICMP ping, reverse DNS and SYN scans. We analyzed some of the data to get an estimation of the IP address...
The Dataset Collection
data
eye 2,996
favorite 1
comment 0
I took the Reddit comment archive and converted all the JSON into one SQLite database using this program that I wrote: https://gist.github.com/ers35/3b615a75fa0ed5e6d5cc I ran a few tests to make sure the number of database rows matches the number of JSON records. "SELECT MAX(rowid) FROM comment" and "SELECT COUNT(id) FROM comment" both return 1659361605. This gives me some confidence as to the integrity of the dataset, but I cannot be 100% sure. The compressed size is 163G....
The Dataset Collection
by NYC Taxi and Limousine Commission
data
eye 1,491
favorite 2
comment 0
FOIA/FOILed Taxi Trip Data from the NYC Taxi and Limousine Commission 2013. Released by http://chriswhong.com/open-data/foil_nyc_taxi/ trip_data.7z and trip_fare.7z are more efficiently compressed versions of the data, you probably want these files. The data is in csv format. For the data files this includes the fields: medallion, hack_license, vendor_id, rate_code, store_and_fwd_flag, pickup_datetime, dropoff_datetime, passenger_count, trip_time_in_secs, trip_distance, pickup_longitude,...
Topics: data, nyc, taxi, fare, csv, FOIA, FOIL
Source: torrent:urn:sha1:6c594866904494b06aae51ad97ec7f985059b135
The Dataset Collection
data
eye 1,040
favorite 0
comment 0
Database of UPC product codes, as compiled by upcdatabase.com
Topics: UPC, Universal Product Code, barcode
The Dataset Collection
data
eye 613
favorite 0
comment 0
Database of UPC product codes, as compiled by upcdatabase.com
Topics: UPC, Universal Product Code, barcode
The Dataset Collection
software
eye 523
favorite 0
comment 0
All the "journal article" DOIs from CrossRef's OAI-PMH server; URLs of just under 50 million journal articles.
Topics: doi, dataset
The Dataset Collection
data
eye 381
favorite 0
comment 0
Database of UPC product codes, as compiled by upcdatabase.com http://sourceforge.net/projects/upcdatabase/
Topics: UPC, Universal Product Code, barcode
The Dataset Collection
by William W. Cohen, MLD, CMU
web
eye 307
favorite 2
comment 0
This dataset was collected and prepared by the CALO Project (A Cognitive Assistant that Learns and Organizes). It contains data from about 150 users, mostly senior management of Enron, organized into folders. The corpus contains a total of about 0.5M messages. This data was originally made public, and posted to the web, by the Federal Energy Regulatory Commission during its investigation. The email dataset was later purchased by Leslie Kaelbling at MIT, and turned out to have a number of...
Topics: Enron, E-mail, Dataset
MusicBrainz Data Dumps
data
eye 155
favorite 0
comment 0
MusicBrainz Database Dump 20141004-015144
Dumps of DISCOGS.ORG Metadata (2008-Present)
by DISCOGS.ORG
software
eye 83
favorite 0
comment 0
This is the monthly dump of DISCOGS.ORG data, provided to the public domain. This dump has been generated and archived automatically. Official name is discogs_20110601 and the data is from 2011-06-01.
MusicBrainz Data Dumps
data
eye 53
favorite 0
comment 0
MusicBrainz Database Dump 20150523-002918
MusicBrainz Data Dumps
data
eye 51
favorite 0
comment 0
MusicBrainz Database Dump 20140329-003047
MusicBrainz Data Dumps
data
eye 51
favorite 0
comment 0
MusicBrainz Database Dump 20140924-014210
MusicBrainz Data Dumps
data
eye 40
favorite 0
comment 0
MusicBrainz Database Dump 20140215-002226
MusicBrainz Data Dumps
data
eye 37
favorite 0
comment 0
MusicBrainz Database Dump 20140705-002713
MusicBrainz Data Dumps
data
eye 33
favorite 1
comment 0
MusicBrainz Database Dump 20150124-024159
MusicBrainz Data Dumps
data
eye 28
favorite 0
comment 0
MusicBrainz Database Dump 20150408-002807
The Dataset Collection
data
eye 27
favorite 0
comment 0
Minecraft Archive Project: MINECRAFTFORUM.NET
MusicBrainz Data Dumps
data
eye 26
favorite 0
comment 0
MusicBrainz Database Dump 20140806-002953
MusicBrainz Data Dumps
data
eye 24
favorite 0
comment 0
MusicBrainz Database Dump 20140611-002358
MusicBrainz Data Dumps
data
eye 23
favorite 0
comment 0
MusicBrainz Data Dump 20120623-002010
Topics: database, music, musicbrainz
MusicBrainz Data Dumps
data
eye 21
favorite 0
comment 0
MusicBrainz Database Dump 20140809-003420
MusicBrainz Data Dumps
data
eye 19
favorite 0
comment 0
MusicBrainz Database Dump 20140726-002617
MusicBrainz Data Dump 20111231-001743 (partial, no edit data included)
MusicBrainz Data Dumps
data
eye 18
favorite 0
comment 0
MusicBrainz Database Dump 20140719-003856
MusicBrainz Data Dumps
data
eye 18
favorite 0
comment 0
MusicBrainz Database Dump 20141220-014845
MusicBrainz Data Dumps
data
eye 17
favorite 0
comment 0
MusicBrainz Database Dump 20140219-002209
MusicBrainz Data Dumps
data
eye 17
favorite 0
comment 0
MusicBrainz Database Dump 20140604-002730
MusicBrainz Data Dumps
data
eye 17
favorite 0
comment 0
MusicBrainz Database Dump 20140716-002400
MusicBrainz Data Dumps
data
eye 16
favorite 0
comment 0
MusicBrainz Database Dump: 20130612-002814
MusicBrainz Data Dumps
data
eye 16
favorite 0
comment 0
MusicBrainz Database Dump: 20131005-002016
MusicBrainz Data Dumps
data
eye 16
favorite 0
comment 0
MusicBrainz Database Dump 20140226-002239
MusicBrainz Data Dumps
data
eye 16
favorite 0
comment 0
MusicBrainz Database Dump 20141001-015012
MusicBrainz Data Dumps
data
eye 16
favorite 0
comment 0
MusicBrainz Database Dump 20141217-015115
MusicBrainz Data Dumps
data
eye 15
favorite 0
comment 0
MusicBrainz Database Dump 20140920-014204
MusicBrainz Data Dumps
data
eye 15
favorite 0
comment 0
MusicBrainz Database Dump 20150304-002406
MusicBrainz Data Dumps
data
eye 15
favorite 0
comment 0
MusicBrainz Database Dump 20160123-004631
MusicBrainz Data Dumps
data
eye 14
favorite 0
comment 0
MusicBrainz Database Dump 20140326-003531
MusicBrainz Data Dumps
data
eye 14
favorite 0
comment 0
MusicBrainz Database Dump 20140419-002346
MusicBrainz Data Dumps
data
eye 14
favorite 0
comment 0
MusicBrainz Database Dump 20140827-002941
MusicBrainz Data Dumps
data
eye 14
favorite 0
comment 0
MusicBrainz Database Dump 20141224-015659
MusicBrainz Data Dumps
data
eye 14
favorite 0
comment 0
MusicBrainz Database Dump 20150121-024325
MusicBrainz Data Dumps
data
eye 13
favorite 0
comment 0
MusicBrainz Database Dump 20140308-002315
MusicBrainz Data Dumps
data
eye 13
favorite 0
comment 0
MusicBrainz Database Dump 20140702-002926
MusicBrainz Data Dumps
data
eye 13
favorite 0
comment 0
MusicBrainz Database Dump 20141105-014932
MusicBrainz Data Dumps
data
eye 13
favorite 0
comment 0
MusicBrainz Database Dump 20141122-015020
MusicBrainz Data Dumps
data
eye 12
favorite 0
comment 0
MusicBrainz Database Dump 20140531-002519
MusicBrainz Data Dumps
data
eye 12
favorite 0
comment 0
MusicBrainz Database Dump 20140628-003321
MusicBrainz Data Dumps
data
eye 12
favorite 0
comment 0
MusicBrainz Database Dump 20140823-003059
MusicBrainz Data Dumps
data
eye 12
favorite 0
comment 0
MusicBrainz Database Dump 20140903-011536
MusicBrainz Data Dumps
data
eye 12
favorite 0
comment 0
MusicBrainz Database Dump 20141108-014733
MusicBrainz Data Dumps
data
eye 12
favorite 0
comment 0
MusicBrainz Database Dump 20150114-015208
MusicBrainz Data Dumps
data
eye 12
favorite 0
comment 0
MusicBrainz Database Dump 20150829-004806
MusicBrainz Data Dumps
data
eye 11
favorite 0
comment 0
MusicBrainz Database Dump: 20130126-014359
MusicBrainz Data Dumps
data
eye 11
favorite 0
comment 0
MusicBrainz Database Dump 20140205-002341
MusicBrainz Data Dumps
data
eye 11
favorite 0
comment 0
MusicBrainz Database Dump 20140222-002157
MusicBrainz Data Dumps
data
eye 11
favorite 0
comment 0
MusicBrainz Database Dump 20140319-002202
MusicBrainz Data Dumps
data
eye 11
favorite 0
comment 0
MusicBrainz Database Dump 20140322-002320
MusicBrainz Data Dumps
data
eye 11
favorite 0
comment 0
MusicBrainz Database Dump 20140709-002346
MusicBrainz Data Dumps
data
eye 10
favorite 0
comment 0
MusicBrainz Database Dump: 20131012-002238
MusicBrainz Data Dumps
data
eye 10
favorite 0
comment 0
MusicBrainz Database Dump 20140312-002422
MusicBrainz Data Dumps
data
eye 10
favorite 0
comment 0
MusicBrainz Database Dump 20140315-002243
MusicBrainz Data Dumps
data
eye 10
favorite 0
comment 0
MusicBrainz Database Dump 20140712-002607
MusicBrainz Data Dumps
data
eye 10
favorite 0
comment 0
MusicBrainz Database Dump 20150103-015415
MusicBrainz Data Dumps
data
eye 10
favorite 0
comment 0
MusicBrainz Database Dump 20150117-024028
MusicBrainz Data Dumps
data
eye 10
favorite 0
comment 0
MusicBrainz Database Dump 20150128-031546
MusicBrainz Data Dumps
data
eye 10
favorite 0
comment 0
MusicBrainz Database Dump 20150214-002534
MusicBrainz Data Dumps
data
eye 9
favorite 0
comment 0
MusicBrainz Database Dump: 20130306-003205
MusicBrainz Data Dumps
data
eye 9
favorite 0
comment 0
MusicBrainz Database Dump: 20130907-001934
MusicBrainz Data Dumps
data
eye 9
favorite 0
comment 0
MusicBrainz Database Dump 20140212-002328